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The  U.S.  Army  Communications-Tilectronics  Tife  Cycle  Management  Command  Software  Engineering  Center  is  working 
with  the  Software  Engineering  Institute  in  creating  a  framework  for  cooperative  government!  industry  appraisals  for  process 
improvement  and  risk  evaluation.  Traditionally,  solicitations  for  Uepartment  of  Defense  projects  have  included  some  sort  of 
risk  evaluation.  Though  risk  evaluations  are  only  one  component  of  a  source  selection,  all  bidders  underwent  the  risk  evalu¬ 
ation  site  visit,  costing  the  government  significant  time  and  effort  in  evaluatingpotentially  several  organisations  that  would  not 
perform  the  work  solicited.  The  concept  developed  is  to  partially  base  an  award  on  the  merits  of  a  process  proposal  with  the 
understanding  that  an  on-site  evaluation  would  follow  after  contract  award.  Another  aspect  of  these  appraisals  is  that  repre¬ 
sentatives  from  the  government  and  the  organisation  being  appraised  work  together  on  the  appraisal  team  to  jointly  evaluate 
the  organisation.  This  article  describes  the  efforts  of  the  authors  in  performing  post-award  cooperative  appraisals,  the  lessons 
learned,  and  the  benefits  to  both  the  government  and  the  appraised  organisation. 


Traditionally,  solicitations  for  Depart¬ 
ment  of  Defense  (DoD)  projects 
have  included  some  method  of  risk  evalu¬ 
ation  to  determine  the  level  of  risk  that 
the  project  manager  (PM)  will  face  in 
selecting  a  bidder  to  provide  the  prod¬ 
ucts/services  for  his  or  her  program.  This 
risk  evaluation  could  take  the  form  of  a 
Software  Capability  Evaluation  (SCE®“),  a 
methodology  developed  by  the  Software 
Engineering  Institute  (SEP“)  at  Carnegie 
Mellon  University,  or  other  methods. 

The  Communications-Electronics  Life 
Cycle  Management  Command  (C-E 
LCMC)  Software  Engineering  Center 
(SEC)  developed  a  streamlined  form  of 
the  evaluation  called  the  Software  Process 
Risk  Evaluation  (SPRE)  that  has  been 
mandated  for  all  major  C-E  LCMC  acqui¬ 
sitions.  This  method,  like  the  SCE,  has 
been  used  during  the  solicitation  process 
to  evaluate  all  potential  vendors  and  pro¬ 
vide  input  to  the  evaluation  factors  for  the 
solicitation. 

There  were  several  problems  with  the 
SCE  and  SPRE  methods.  There  often  was 
not  enough  time  to  prepare  for  the  evalu¬ 
ation.  A  lack  of  historical  information 
about  the  bidders’  processes  made  the 
evaluations  more  critical  while  decreasing 
the  effectiveness  of  a  short,  intense  on¬ 
site  visit.  The  cost  of  the  evaluations  were 
high  for  both  the  government,  who  has  to 
visit  all  of  the  bidders  during  the  propos¬ 
al  evaluation  period,  and  the  contractor, 
who  has  to  apply  significant  resources  to 
prepare  data  and  provide  people  to  be 
interviewed  during  the  on-site  visit.  Both 
parties  potentially  spend  money  to  sup¬ 
port  multiple  source  selection  evaluations. 
The  government  expends  resources  to 
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evaluate  the  losing  contractors,  and  the 
contractor  could  possibly  have  to  support 
multiple  evaluations  in  a  given  timeframe. 

Acquisition  Reform 

Industry  has  matured  over  the  years.  The 
SEI  issued  the  Capability  Maturity 
Model®  for  Software  (SW-CMM®)  in 
1993.  Since  that  time,  many  of  the  orga¬ 
nizations  that  bid  on  acquisition  solicita¬ 
tions  have  undergone  process  improve¬ 
ment  initiatives  using  SW-CMM  or  its 
successors,  including  the  CMM  Inte¬ 
gration®"  (CMMI®).  With  the  advent  of 
the  CMMI,  a  new  appraisal  method  -  the 
Standard  CMMI  Appraisal  Method  for 
Process  Improvement  (SCAMPI®")  [1]  — 
has  been  developed  and  is  quickly  becom¬ 
ing  the  appraisal  method  of  choice, 
regardless  of  which  CMM  is  being  used 
as  the  model  for  process  improvement. 
SCAMPI  can  be  used  in  lieu  of  the 
SCE/ SPRE  for  evaluating  bidders  during 
a  source  selection. 

Government  acquisition  reform  has 
evolved  as  well.  The  Interim  Defense 
Acquisition  Guidebook,  paragraph 
C5.2.3.5.6.1.5,  states  the  following: 

Select  contractors  with  domain 
experience  in  developing  compara¬ 
ble  software  systems;  with  success¬ 
ful  past  performance;  and  with  a 
mature  software  development  capa¬ 
bility  and  process.  Contractors  per¬ 
forming  software  development  or 
upgrade(s)  for  use  in  an  [Acqui¬ 
sition  Category]  ACAT  I  or  ACAT 
lA  program  shall  undergo  an  evalu¬ 
ation,  using  either  the  tools  devel¬ 
oped  by  the  [SEI]  or  those 
approved  by  both  the  DoD 
Components  and  the  Deputy 
Director,  Software  Intensive  Sys¬ 


tems.  At  a  minimum,  full  compli¬ 
ance  with  SEI  Capability  Maturity 
Model  Level  3,  or  its  equivalent  in 
an  approved  evaluation  tool,  is  the 
department’s  goal.  However,  if  the 
prospective  contractor  does  not 
meet  full  compliance,  risk  mitiga¬ 
tion  planning  shall  describe,  in 
detail,  the  schedule  and  actions  that 
win  be  taken  to  remove  deficiencies 
uncovered  in  the  evaluation  process. 
Risk  mitigation  planning  shall 
require  Product  Manager  approval. 
The  Deputy  Director,  Software 
Intensive  Systems  shall  define  Level 
3  equivalence  for  approved  evalua¬ 
tion  tools.  The  evaluation  shall 
examine  the  business  unit  proposed 
to  perform  the  work.  The  reuse  of 
existing  evaluation  results  per¬ 
formed  within  a  2-year  period  prior 
to  the  date  of  the  government  solic¬ 
itation  is  encouraged.  [2] 

Later  guidance  clarified  Level  3  equiv¬ 
alence.  The  Software  Development 
Capability  Evaluation  [3],  developed  by 
the  U.S.  Air  Force  with  approved  core  set 
revisions,  was  determined  to  be  an  accept¬ 
able  alternative  set  of  criteria  to  the  SW- 
CMM.  The  use  of  CMMI  Systems 
Engineering  and  Software  Engineering 
Vers.  1.1  Level  3  criteria  is  another  accept¬ 
able  alternative  to  the  use  of  SW-CMM 
Level  3  criteria. 

Newer  guidance  from  the  Office  of 
the  Secretary  of  Defense/Acquisition, 
Technology,  and  Logistics  no  longer 
requires  Level  3  equivalent  ratings. 
Process  improvement  is  encouraged,  but 
maturity  levels  are  no  longer  stated  in  the 
guidance.  The  rationale  is  that  many 
organizations  have  reached  maturity 
Level  3  or  better,  and  that  the  original 
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direction  was  not  to  have  organizations 
go  for  the  rating,  but  to  initiate  organiza¬ 
tional  process  improvement. 

The  CMMI,  in  its  continuous  represen¬ 
tation,  allows  an  organization  to  be 
appraised  to  determine  a  process  capabili¬ 
ty  profile;  the  organization  can  look  at 
Process  Areas  (PAs)  of  interest  based  on 
organizational  goals  and  see  how  well  they 
have  implemented  them.  This  can  also 
allow  a  PM  to  focus  on  areas  that  are  of 
high  risk  for  a  particular  acquisition,  rather 
than  look  at  overall  maturity  across  all  PAs. 

Appraisal  Reuse 

To  reduce  costs  and  resources  associated 
with  evaluating  development  capabilities, 
the  DoD  is  working  with  industry  to  pro¬ 
vide  implementing  mechanisms  that  better 
support  the  reuse  of  appraisals.  Results  of 
previous  appraisals  conducted  on  the  orga¬ 
nizational  unit  proposing  to  do  the  work 
may  be  an  acceptable  alternative  to  the 
government  performing  a  new  appraisal. 
The  reused  appraisals  must  be  shown  to  be 
independent,  i.e.,  at  a  minimum,  the  lead 
appraiser  should  not  be  from  within  the 
organization  being  appraised. 

C-E  LCMC  Acquisition  Strategy 

The  SEC  has  been  involved  in  the  exami¬ 
nation  of  cooperative  government/indus¬ 
try  appraisals  as  an  alternative  to  the  con¬ 
ventional  acquisition  strategy  for  several 
years  [4].  The  SEC  took  the  initiative  in 
working  with  the  acquisition  center  at 
Fort  Monmouth  to  identify  software¬ 
intensive  acquisitions  occurring  there. 
The  interim  project  office  was  contacted 
and  cost-effective  strategies  developed  to 
address  the  acquisition  agent’s  need  to 
reduce  risk.  One  of  these  strategies  was  to 
wait  until  after  contract  award  and  per¬ 
form  a  post-award  appraisal  of  the  suc¬ 
cessful  bidder. 

To  make  the  post-award  appraisal 
viable,  the  solicitation  package  must  have 
the  appropriate  language.  The  contract 
should  require  that  the  product  be  devel¬ 
oped  using  CMM/CMMI  Level  3 
processes.  Proof  is  to  be  submitted, 
demonstrating  that  the  contractor  is  rated 
as  a  Level  3  organization  (e.g,  copies  of 
CMM/CMMI  appraisals  and  process 
improvement  track  record).  If  the  organi¬ 
zation  cannot  verify  Level  3,  a  detailed 
process  improvement  plan,  including  a 
schedule  that  leads  to  a  Level  3  appraisal 
is  submitted.  This  provides  the  acquisi¬ 
tion  agent  with  material  in  which  to  eval¬ 
uate  the  bidders’  process  maturity  during 
the  source  selection  without  the  effort 
required  for  on-site  visits. 

The  post-award  appraisal  must  also  be 


contained  in  the  contract.  Typically,  C-E 
LCMC  requires  that  the  contractor  is 
responsible  for  leading  and  conducting  an 
appraisal  with  government  participation 
on  the  appraisal  team.  This  is  appropriate 
as  it  is  the  contractor’s  plant  and  process¬ 
es.  The  contractor  is  required  to  submit  an 
appraisal  plan  or  process  proposal  as  a 
formal  deliverable  after  the  contract  is 
awarded. 

If  a  CMMI  SCAMPI  appraisal  is 
selected  or  required,  additional  options 
present  themselves.  The  SCAMPI  frame¬ 
work  defines  three  classes  of  appraisal.  A 
Class  C  appraisal  is  a  very  cursory  look  at 
the  processes  of  the  organization.  It  can 
often  be  nothing  more  than  a  review  of 
process  documentation  and  its  application 
to  the  project.  A  Class  B  appraisal  is  more 

*^The  CMMI,  in 
its  continuous 
representation,  allows 
an  organization  to  be 
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organizational  goals  and 
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robust,  but  does  not  emphasize  the  depth 
of  coverage  and  rigor  that  result  in  a 
maturity  level  rating  for  the  organization. 
Class  A  appraisals  are  performed  by  SEI- 
authorized  SCAMPI  lead  appraisers.  They 
can  lead  to  a  formal  maturity  rating  or 
capability  profile  that  is  submitted  to  the 
SET  The  government  determines  which 
class  SCAMPI  appraisal  is  appropriate  for 
the  acquisition. 

With  the  SEI  developing  formal  mech¬ 
anisms  for  SCAMPI  B  and  C  appraisals, 
this  methodology  is  fitting  into  a  DoD 
approach  suggested  by  Mark  Schaeffer, 
the  DoD  sponsor  for  CMMI,  and 
Director,  Systems  Engineering  for  the 


Office  of  the  Secretary  of  Defense.  This 
approach,  reported  by  Mike  Phillips  in 
“CMMI:  A  Progress  Report”  [5],  would 
have  acquirers  look  for  strengths  or  weak¬ 
nesses  in  the  development  processes  that 
constitute  risks  to  the  proposed  develop¬ 
ment  effort.  Satisfying  CMMI  goals, 
process  areas,  or  maturity  or  capability  lev¬ 
els  would  not  be  the  point;  it  is  primarily  a 
risk  identification  and  mitigation 
approach. 

Schaeffer  suggests  a  three-phased 
approach.  In  phase  1,  bidders  would  be 
appraised  for  PAs  that  the  Government 
Program  Office  considers  highest  risk. 
After  contract  award,  in  phase  2,  the  win¬ 
ning  team  undergoes  a  baseline  appraisal, 
using  a  risk-based  analysis  of  the  process 
strengths  and  weaknesses,  thereby  estab¬ 
lishing  action  plans  for  future  checks.  In  a 
risk-based  appraisal  framework,  the 
CMMI  is  used  to  identify  and  group  weak¬ 
nesses  to  address  systemic  problems.  The 
CMMI’s  process  categories  -  Project 
Management,  Engineering,  Support,  and 
Process  Management  -  may  be  used  to 
define  areas  that  the  PM  believes  most 
affects  the  developer  team’s  contract  per¬ 
formance.  Using  these  categories  effec¬ 
tively  covers  both  product  and  process 
risk.  In  phase  3,  the  risks  are  monitored  to 
closure. 

Tying  CMMI/risk  mitigation  to  an 
award  fee  on  a  contract  vehicle  can  be  a 
good  incentive  for  the  contractor.  A  pro¬ 
gram  office  should  consider  wording  the 
contract  to  have  a  continuous  process 
monitoring  function.  For  example,  a  PM 
could  elect  to  utilize  the  Defense  Contract 
Management  Agency  (DCMA)  or  some 
other  matrix  organization  to  perform 
process  monitoring.  This  is  also  a  recom¬ 
mendation  made  in  the  Workshop  on 
CMMI  Use  in  Acquisition  [6]. 

Benefits  of  Post-Award 
Appraisals 

The  PM  saves  resources  by  limiting  the 
number  of  appraisals  required  for  a  solic¬ 
itation.  Maturity  risk  is  only  one  of  many 
evaluation  factors  for  an  acquisition 
source  selection  evaluation.  The  increased 
risk  of  not  performing  an  on-site  evalua¬ 
tion  for  each  bid  is  mitigated  by  the  cost 
and  time  savings.  Many  organizations  have 
been  performing  process  improvement 
initiatives  for  more  than  10  years.  This 
legacy  of  process  improvement  tends  to 
ensure  that  processes  are  defined,  which  is 
the  focus  of  maturity  Level  3  in  the 
CMM/CMMI. 

The  bidders  benefit  in  this  approach  as 
well.  Organizations  can  reuse  their 
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Alternate  Mixes  for  CMMI 


Attribute 

SCE/SPRE/SCAMPI 

Cooperative  Appraisal 

Timeframe 

■  Pre-award. 

■  May  be  used  post-award  for 
contract  monitoring. 

■  Pre-award  or  post-award. 

■  Recommend  post-award 
baseline  and  follow-on  for 
contract  monitoring. 

■  Can  be  linked  to  supplier’s 
process  improvement  appraisal 
schedule. 

Cost 

■  High  (both  supplier  and 
acquirer). 

■  Lower  (cost  sharing). 

Cost  Effectiveness 

■  Low,  especially  if  supplier  must 
support  evaluations  for  losing 
solicitation. 

■  High. 

■  Eliminates  cost  of  evaluating 
losing  bidders. 

Resources  Needed 

■  Resource-intensive. 

■  Resource  sharing  for  staffing 
appraisals. 

■  Supplier  can  reuse  process 
data. 

Incentive  for  Process 
Improvement 

■  Little  if  no  contractual  language 
for  process  improvement. 

■  Supplier  can  reuse  process 
assets  developed  for  process 
improvement. 

■  Potential  for  appraisal  reuse  on 
subsequent  solicitations. 

Integrated  Product 
Development 

■  Government-only  team  does  not 
form  integrated  teams  with 
supplier. 

■  Fosters  early  development  of 
integrated  teams. 

■  Facilitates  government/supplier 
communication. 

Risk 

■  Mitigates  selection  of  high-risk 
bidders. 

■  Does  not  allow  for  an  in  depth 
risk  evaluation  of  the  supplier 
unless  applied  post-award. 

■  Early  risk  mitigation  if  performed 
early  in  contract  execution. 

■  Continuous  risk  monitoring  and 
control  if  used  for  contract 
monitoring. 

Table  1 :  Evaluation  and  Cooperative  Appraisal  Attributes 


process  data  for  multiple  solicitations. 
They  do  not  have  the  expense  of  prepar¬ 
ing  for  an  on-site  visit  for  a  solicitation 
that  they  are  not  guaranteed  to  win. 
Cooperative  appraisals  reward  the  organi¬ 
zation  for  its  process  improvement 
efforts,  thereby  encouraging  internal 
process  improvement  activities.  Table  1 
summarizes  the  attributes  and  how  con¬ 
ventional  evaluations  compare/ contrast  to 
the  cooperative  appraisal  methodology. 

Pilot  Effort  for  Cooperative 
Appraisals 

The  SEC  was  able  to  participate  in  an 
ACAT  I  acquisition  where  the  PM  was 
consulted  before  the  release  of  the 
Statement  of  Work  (SOW).  After  being 
briefed  on  the  benefits  of  performing  a 
post-award  appraisal,  the  PM  agreed  to 
have  language  added  to  the  SOW  for  a 
CMMI  self-assessment  of  all  major  con¬ 
tributors  to  the  product  development 
effort  using  the  SCAMPI  method  with  up 
to  four  government  participants. 

The  winning  contractor  team  submit¬ 
ted  an  assessment  plan  to  have  four  team¬ 
mates  undergo  a  self-assessment.  The 
original  plan  called  out  in  the  SOW  was  to 
submit  the  plan  within  60  days  after  con¬ 
tract  award  and  perform  the  self-assess¬ 
ments  within  120  days  after  contract 
award.  The  assessment  would  establish  a 
process  capability  baseline  on  the  organi¬ 
zational  unit  doing  the  work  on  the  pro¬ 
gram  as  well  as  determine  risk  areas  to  be 


monitored  over  the  acquisition  life  cycle. 
Since  the  contract  was  awarded  to  a  team 
of  contractors,  consideration  was  given  to 
the  timing  requirements  for  both  the  plan 
and  the  conduct  of  the  self-assessments. 
The  SOW  provided  a  list  of  CMMI  PAs  of 
interest.  These  were  the  minimum  set  of 
processes  that  were  to  be  assessed,  which 
tied  into  the  program  office’s  key  perfor¬ 
mance  parameters,  goals,  and  objectives 
for  the  program. 

Using  multi-organizational  teams 
brings  a  new  risk  to  the  conventional 
acquisition  evaluation  methodology.  The 
contracting  team  may  have  individually 
performed  process  improvement  efforts, 
but  the  team  does  not  necessarily  share 
instimtionalized  processes.  Former  assess¬ 
ments  are  not  directly  applicable  to  the 
newly  formed  team  and  are  less  predictive 
of  the  maturity  or  capability  of  the  team 
doing  the  work. 

In  this  pilot,  the  prime  contractor 
worked  to  develop  capstone  processes 
that  aU  teammates  would  follow,  but  addi¬ 
tionally  allowed  for  the  individual  organi¬ 
zation’s  processes  to  be  used.  For  exam¬ 
ple,  a  project  Software  Development  Plan 
(SDP)  was  written  by  the  prime,  which 
called  out  use  of  the  other  organizations’ 
SDPs  for  their  software  contributions  to 
the  project. 

The  contractor’s  self-assessment  plan 
called  for  self-assessments  to  be  held  at 
multiple  organizations.  Each  self-assess¬ 
ment  covered  aU  maturity  Level  2  and  3 


PAs.  Some  sites  added  higher  maturity 
PAs.  The  government  worked  with  the 
contractor  team  to  try  to  leverage  each 
organization’s  internal  process  improve¬ 
ment  activities.  AU  of  the  contractors  were 
transitioning  from  SW-CMM  to  CMMI; 
they  were  already  implementing  organiza¬ 
tional  process  improvement  plans. 

Where  feasible,  the  government 
allowed  the  organization  to  add  the  acqui¬ 
sition  project  to  the  Ust  of  internal  pro¬ 
jects  being  appraised  to  avoid  unneces¬ 
sary  effort  of  preparing  for  an  indepen¬ 
dent  assessment  of  one  program.  For  at 
least  one  organization,  this  meant  sched- 
uUng  the  appraisal  as  part  of  their  exter¬ 
nally  led  SCAMPI.  This  benefited  the 
government  as  well,  since  the  project  was 
not  yet  fully  under  way  and,  therefore, 
many  PAs  (e.g.,  technical  solution,  verifi¬ 
cation,  and  vaUdation)  could  not  be  rigor¬ 
ously  assessed  -  the  project  simply  was 
not  to  that  point  in  its  Ufe  cycle.  The  addi¬ 
tion  of  projects  in  different  stages  of 
maturity  provided  a  more  rounded  picture 
of  the  organization  and  how  its  institu- 
tionaUzed  processes  would  be  appUed  to  a 
future  project. 

The  SEC  provided  the  two  authors  as 
the  core  team  that  participated  on  the  self- 
assessments.  Using  available  local  DCMA 
representatives  augmented  government 
participation.  The  PM  had  contracted 
with  DCMA  to  perform  process  monitor¬ 
ing  over  the  course  of  the  development 
effort,  so  their  participation  in  the  self- 
assessments  acted  as  a  kick-start  in  under¬ 
standing  the  details  of  the  project’s 
processes. 

Cooperative  Appraisals  and 
the  CMMI 

The  way  that  acquisition  reform  is  evolv¬ 
ing  is  similar  to  that  of  the  evolution  from 
SW-CMM  to  CMMI.  CMMI  provides 
integration  of  software  with  other  disci- 
pUnes,  i.e.,  systems  engineering  and  inte¬ 
grated  product  and  process  development 
(IPPD),  manifested  in  integrated  product 
teams  (IPTs). 

There  is  an  analogous  evolution  in 
acquisition  —  a  paradigm  shift  from  the 
traditional  buyer-seUer  relationship  to  that 
of  an  IPT  where  both  the  government 
and  contractor  share  responsibiUty  for  the 
end  product.  This  integrated  team  con¬ 
cept  necessitates  a  team  approach  to 
assessment  as  well.  In  a  cooperative 
appraisal,  the  government  participants  add 
objectivity  and  diverse  experience.  The 
team  that  is  formed  during  cooperative 
appraisals  can  be  extended  to  established 
IPTs  for  the  acquisition  life  cycle,  where 
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government  participants  in  the  assessment 
can  also  be  members  of  the  government’s 
project  management  team. 

Anyone  who  has  participated  in  an 
SEI  appraisal  can  attest  to  the  fact  that 
the  intensity  of  the  shared  experience 
does  much  to  foster  team  building.  A 
foundation  of  mutual  respect,  equality, 
and  cooperation  is  established  through¬ 
out  an  appraisal.  These  sentiments  can  be 
brought  back  to  the  project  management 
office  and  used  to  continuously  facilitate 
communication  between  the  government 
and  contractor.  The  cooperative 
appraisal  process  allows  both  parties  to 
utilize  and  benefit  from  the  appraisal 
data;  there  is  open  process  communica¬ 
tion  between  the  government  and  their 
contractor.  This  gives  the  government  an 
understanding  of  the  way  the  contractor 
does  business. 

The  government  gets  to  meet  the  key 
players,  from  the  president  of  the  corpo¬ 
ration  to  the  practitioners  at  the  develop¬ 
er’s  site,  and  may  additionally  see  the  facil¬ 
ities  and  operations  for  the  project.  This 
interaction  establishes  positive  working 
relationships  and  provides  a  greater 
understanding  of  what  the  government/ 
contractor  IPT  brings  to  the  program. 

In  a  cooperative  appraisal,  the  govern¬ 
ment  leverages  the  project  and  process 
expertise  brought  by  the  contractor,  thus 
facilitating  the  government’s  assessment 
efforts.  Usually  there  are  appraisal  team 
members  that  are  part  of  the  contractor’s 
process  improvement  group  or  belong  to 
the  team  working  on  one  of  the  projects 
being  appraised.  These  people  can  quickly 
guide  the  government  appraisal  team  mem¬ 
ber  to  the  appropriate  process  artifacts  or 
answer  appraisal-focused  questions. 

Lessons  Learned 

There  was  some  trepidation  on  the  part  of 
the  contractors  in  performing  the  cooper¬ 
ative  appraisal.  It  was  initially  perceived 
that  the  government  representatives  were 
coming  in  to  perform  an  audit  or  evalua¬ 
tion,  not  to  participate  in  an  internal 
process  appraisal.  Discussions  with  the 
site  coordinator  and  contractor  manage¬ 
ment  before  the  appraisal  helped  over¬ 
come  these  perceptions. 

The  government  representatives 
explained  that  the  cooperative  appraisals 
were  one  more  manifestation  of  the  IPT 
method  of  running  a  program  where  the 
government  and  the  contractor  share 
responsibility  for  the  program’s  success. 
The  contractors  were  reminded  that  their 
organization’s  formal  appraisal  teams  are 
often  composed  of  personnel  external  to 
the  organization,  and  that  the  government 


representatives’  goal  was  to  be  integral 
members  of  the  team.  The  measure  of 
success  would  be  how  forthcoming  those 
interviewed  were  during  the  self-assess¬ 
ment.  Since  many  of  the  people  inter¬ 
viewed  had  experience  with  internal 
appraisals,  where  they  may  speak  with 
people  outside  of  their  business  unit  or 
external  people,  the  cooperative  appraisal 
team  did  not  experience  any  difficulties  in 
this  area. 

Internal  process  improvement 
requires  senior  management  commitment. 
The  cooperative  appraisal  teams  usually 
are  comprised  of  several  process  group 
members.  These  process  group  personnel 
are  responsible  for  recommending  the 
future  direction  of  process  improvement 
efforts  in  the  organization.  The  organiza¬ 
tions  being  appraised  were  transitioning 
from  CMM  to  CAIMI;  some  had  never 
piloted  a  CMMI  appraisal.  This  provided 
an  opportunity  for  the  process  group  to 

the  authors  con 
state  that  an  extremely 
beneficial  byproduct 
of  being  on  on 
appraisal  team  is 
that  you  can  learn 
of  many  practices 
that  con  be  adapted 
for  use  in 

your  organization/* 

gather  valuable  information  on  the  state 
of  the  organization  and  how  successfully 
the  transition  efforts  were  proceeding. 
The  external  inputs  from  the  government 
can  assist  the  process  group  by  providing 
an  unbiased  view  of  the  organization,  and 
help  influence  senior  management  in 
determining  what  areas  the  organization 
should  be  concentrating  on  in  their  future 
process  improvement  initiatives. 

The  government  gained  insight  into 
the  different  cultures  of  the  organizations 
comprising  the  contractor  team.  They  had 
different  ways  of  doing  business,  different 
vocabularies,  and  different  ways  of  work¬ 
ing  with  the  prime  contractor  and  the 
government.  Understanding  these  differ¬ 
ences  aids  in  facilitating  communications 
among  the  members  of  the  IPTs  and 


avoids  misinterpretations. 

Having  a  core  team  of  government 
participants  on  all  of  the  appraisals  pro¬ 
vided  continuity.  The  same  personnel  can 
compare  and  contrast  how  the  different 
organizations  are  performing  process 
improvement  and  satisfying  the  practices 
of  the  CMMI.  Common  areas  of  weak¬ 
ness  and  interpretation  issues  can  be 
raised  and  addressed  to  better  support  the 
program. 

If  the  government  provides  several 
people  for  a  cooperative  appraisal,  they 
should  represent  the  major  disciplines 
(e.g.,  program  management,  systems  engi¬ 
neering,  software  engineering,  and  logis¬ 
tics)  that  are  involved  in  the  acquisition.  A 
multi-disciplinary  team  can  cover  a  broad¬ 
er  range  of  PAs,  while  a  given  government 
team  member  can  specialize  in  his  or  her 
area  of  expertise.  Involving  multiple  disci¬ 
plines  allows  the  government  to  examine 
the  developers’  processes,  taking  into  con¬ 
sideration  the  entire  program  life  cycle. 
This  enables  downstream  risks  to  be  iden¬ 
tified  early  in  the  acquisition.  These  risks 
can  be  mitigated  with  less  effort  as  a  result 
of  the  early  collaboration  and  expertise 
among  the  government  appraisal  team 
and  the  developers’  project  teams. 

PMs  considering  using  cooperative 
appraisals  should  enlist  direct  or  matrix 
support  from  the  PM  office  for  participa¬ 
tion  on  the  appraisal  team.  The  detailed 
view  of  the  organization  is  extremely 
valuable  to  a  person  who  is  supporting  the 
PM  in  managing  the  acquisition.  The 
appraisal  team  member  gets  to  meet  and 
speak  with  many  of  the  people  perform¬ 
ing  the  project  work,  interview  senior 
management,  and  understand  their  com¬ 
mitment  to  the  project. 

As  participants  of  several  appraisals, 
the  authors  can  state  that  an  extremely 
beneficial  byproduct  of  being  on  an 
appraisal  team  is  that  you  can  learn  of 
many  practices  that  can  be  adapted  for  use 
in  your  organization.  No  organization  has 
a  monopoly  on  best  practices.  The  more 
exposure  that  you  have  to  the  industry,  the 
more  you  can  recognize  there  are  superior 
ways  of  doing  things. 

Aftermath 

The  joint  appraisals  were  considered  to 
be  a  success  by  both  the  government  and 
industry.  Two  of  the  teammates  who  had 
been  appraised  invited  the  government 
representatives  back  for  a  follow-on 
appraisal  where  the  company  was  trying 
to  achieve  a  higher  maturity  level.  These 
organizations  valued  the  government’s 
contributions  to  the  appraisal  effort,  and 
wished  to  maintain  the  same  experienced 
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appraisal  team  in  this  follow-on  appraisal. 
The  government,  in  turn,  appreciated  the 
opportunity  to  follow  up  on  re-appraising 
the  organizations.  This  allowed  the  gov¬ 
ernment  the  ability  to  witness  the  effect  of 
the  process  improvements  made  in  the 
organizations  since  the  self-assessment.^ 
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